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o ; Abstract 

^' In ordinal symbolic dynamics, transcripts describe the algebraic relationship between ordinal patterns. 

p i' Using the concept of transcript, we exploit the mathematical structure of the group of permutations to de- 



rive properties and relations among information measures of the symbolic representations of time series. 
These theoretical results are then applied for the assessment of coupling directionality in dynamical sys- 



\^ ■ tems, where suitable coupling directionality measures are introduced depending only on transcripts. These 

novel measures estimate information flow in lower space dimension and reduce to well-established cou- 



pling directionality quantifiers when some general conditions are satisfied. Furthermore, by generalizing 
the definition of transcript to ordinal patterns of different lengths, several of the commonly used information 
directionality measures can be encompassed within the same framework. 
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I. INTRODUCTION 



The study of dynamical behavior in interacting complex systems is relevant in different fields 
of science 111 |2|]. Developments in the area of non-linear dynamics and the use of information 
theoretic approaches have greatly contributed to the understanding of ubiquitous phenomena like 
synchronization B and collective behavior in spatially extended systems y, |5|]. Great attention 
has recently been paid to the study of causality and the assessment of coupling directionality in 
dynamical systems la-la]. Granger causality QlOl ] was probably the first method which introduced 
the notion of predictability to detect interaction asymmetry in linear models. Using the concept 
of Granger causality other directionality measures were proposed to account for non-linear inter- 



actions in dynamical systems] 



12h . Apart from the traditional methods based on information 



theoretic concepts M, M, \lA \1M, other authors have suggested the use of non-linear state space 



reconstruction [|6|] and the phase-slope of cross spectra Oil]. The characterization and detection 
of information flow has also been investigated from the viewpoint of ordinal symbolic dynamics 
II15I1 . Several approaches have been proposed suggesting advantages of the use of ordinal symbolic 



dynamics like computational efficiency and robustness against noise [116 



m 



Ordinal time series analysis is a particular form of symbolic analysis whose "symbols" are 
ordinal patterns of a given length L > 2. This concept was introduced by C. Bandt and B. Pompe 
in their seminal paper pOd. in which they also introduced permutation entropy as a complexity 
measure of time series. Since then, ordinal time series analysis has found a number of interesting 
applications in biomedical sciences, physics, engineering, finance, statistics, etc. One important 
aspect of this new tool in data analysis is the fact that the ordinal patterns of length L, which can 
be identified with permutations of L objects, have a well-known mathematical structure. Indeed, 
permutations build a (non-commutative) multiplicative group called the symmetric group of order 
L. The mathematical structure of the symmetric group is exploited by the concept of transcript. 
Transcripts were introduced in [|2l|l and applied for characterizing the synchronization behavior of 
two coupled, chaotic oscillators. In this work we will present a further application, this time to the 
characterization of the coupling directionality between time series. 



II. THEORETICAL SETTING 

Let (jc„)„gNo be a sequence whose elements x,, belong to a set endowed with a total ordering 
<. The L-block x^ ~ = Xn,Xn+T, ■■■,Xn+(L-i)j can be associated to the ordinal L-pattern n = 
{no,...,n(L-i)) as follows, 

where in case Xj = Xj, we agree to set x, < Xj if, say, i < j. Here, T > 1 is a time delay used for the 
construction of ordinal patterns. Therefore, an ordinal L-pattem (or ordinal patterns of length L) 
is the permutation of the integer numbers 0, 1,..., L - 1 indicating the rank ordering (according to 
their size) of the elements Xn, Xn+r, ..., Xn+{L-i)T, where n is arbitrary, T > 1, and L>2. Specifically, 
n = {no, ..., n{L-\)) may be identified with the permutation ? h^ tt,, < ? < (L - 1). 

The set of ordinal L-pattems forms a finite non-Abelian group of order L! (the so-called sym- 
metric group Sl), when equipped with the product of permutations defined as 

noo- = {o-„Q,c^„^,...,o■„^_^), (1) 

with the inverse element being given by 

n~^ =o(7ro,...,7TL-l), 

and the unity by the identity permutation, 

/J = <0, 1,...,L-1). 

Here, o denotes the sorting operation. For example o(2, 0, 1) = (1,2,0). 

The algebraic structure of Sl is exploited by the concept of transcripts. In fact, being Sl a 
group, given a,/3 e Sl, there always exists a unique r = Tap e Sl, called transcript from the 
source pattern a to the target pattern [i, such that 

Toa=l3, (2) 

where t o a = (ar^, an , —, (^tl^i) (see Eq. ([T]))- It follows that r is a transcript from a to jS if and 
only if T~' is a transcript from /3 to a. As usual, we will write hereafter the product of a and/? just 
as a/3, unless otherwise convenient. As the source pattern a and the target pattern /3 vary over Sl, 
their transcript varies according to Tap = fio or"' . Note that different pairs (a,/3) can share the same 



transcript. More generally, given t e Sl, there exist L! pairs ia,/3) e SlX Sl such that t is the 
transcript from a to /3. Two trivial properties of the transcripts are 

fjS.Q- = (Ta,fl) (3) 

and 

T/3,yT„,^ = yfi-^fia-^ = fiy-^ = t„,^. (4) 

which imp 



see nil, 



ies the transitivity of the transcription operation. For more properties of the transcripts, 
1221]. 

Consider two stationary time series {jcj, {yj. In turn, they provide two sequences of L-ordinal 
patterns, {at} and {/3k}, respectively. Let ;?[(«) {p\{P)) be the probability for the source (target) 
L-pattem a ifi) to occur in {au} ({AD' ^^id pj^(a,l3) the joint probability. Then, the probability 
function of the transcripts, Piir), r e Sl, can be written as 

Pl(r) = X ^^^^'^)' 

(ff,/3):/3a-'=r 

Thus, the entropy of the joint probability function pj^ and the entropy of the corresponding tran- 
script probability function pj^ are defined as 

H(a,j3) = - Yj piia,mogpi(a,fi), 

afieSt 

and 

H{T) = -Y,Pl(j)\0gpl{T), 

reSt 

respectively, where we have used H{a,p) = H{pj) and H(t) = H(pJ^) for notational convenience. 
The definition of transcripts given by Eq. <^, provides the algebraic relationship between 
source and target ordinal patterns. It follows that, given the triple (a,/3,T), the knowledge of 
any pair of symbols, i.e. (a,/3), (a, t), or (J3, t), univocally determines the remaining symbol. This 
important property implies 

H{a,/3) = Hia,T) = HQ3,T). (5) 

More general, given the random variables a", 1 < n < A^^, with outcomes in Sl, then 

//(..., a'",ff""'\...) =^(...,q:",v,,„«i,...) = //(..., a", v,+i,„«,...) (6) 

= //(..., Ta'\a"+i-,<^"^ , •••) = H(..., Tan+la", a"'^ , ■■■) (J) 



because any of the random variable pairs explicitly shown in ©-([T]) can be determined from any 
other variable pair. 



The concept of coupling complexity was first introduced in [ 1221 ] along with two complexity 
indices for its quantification. Coupling complexity refers to the relationship among dynamical 
system components; in general, it differs from the complexity of the individual components or 
from their sum. Here, we consider only one of two coupling complexity indices proposed, namely 

C{a,p) = mm{Hia), H(J3)} - {H{a,P) - H{r)). (8) 

By means of Eq. dS]), C{a,/3) can be written as 

C(a,/3) = mm{Iia,T),I(J3,T)}, (9) 

where / denotes mutual information. As mutual information is a positive definite quantity, we 
demonstrated here again that C(a,/3) > 0. The complexity index C(a,/3) can also be written as 

Cia,p) = H{t) - max{//(Qr | [3), H(J3 \ a)}, (10) 

where H(a | jS) is a conditional entropy. Since C(a,/3) > 0, Eq. (ITOl) implies H(t) > max{H(a \ 
/3), H(J3 I a)}. The complexity can be generalized to multivariate time series analysis by means of 
the following expression 

da' ,a\..., a'«) = mm{Hia'), H{a\ . . . , //(«'")} + H{Tn, T23, • • • , T(,„.i)„,) - H{a' ,a\..., a'") 

(11) 
C{a\a^, ..., a'") = min /(a'; T12, T23, . . . , T(,„-i)™), (12) 

1 <i<m 

where a" denotes the symbolic representation of the n''' time series and T(„-i)„ are the transcripts 



connecting symbolic representations a" ' and a". A proof of (fT2l) is presented in [|23ll . Similarly 
to the bivariate case, the generalized coupling complexity is invariant under the interchange of the 
a'"s. For instance, consider three symbolic representations {7,}, {y3,}, and {or,}, and all possible 
transcripts {(r-^,/;);}, {{Ty^a)i], and {(rg^Q,),}. Since given two of the three transcripts Ty^, Ty^a, and 
Tfj^a the third one can be determined via (|3]) and ©, it follows that Hijy^p, Ty^a) = H{Ty^fj, Tfj^a) = 
H(Ty^a, T^,o) and therefore the invariance of C{a^^y) (see Eq. (1721) ) under permutation of its argu- 



ments. For a general proof of this property see [|23ll 



III. INFORMATION DIRECTIONALITY 



A. Methods 



The detection of the coupling direction between dynamical systems requires asymmetric mea- 
sures sensitive to the part of information not contained in the joint past of the systems. The 
conditional mutual information (CMI) is such a quantity, having been already used in several ap- 



plications 11 141 12411. We will consider the CMI within the framework of ordinal symbolic dynamics 



as already proposed in different approaches fll6Ul7ll . First, we generate symbolic representations 



and transcripts for coupled dynamical systems using length L and delay T. Let {a,}, {yS,}, {y,} be 
three symbolic representations. The CMI can be written as follows 

I{y,P I a) = H(y \ a) - H(y \ /3, a). (13) 

For {y,} = {ai+A], with A > 0, Eq. (fT3l) becomes a measure of coupling directionality between two 



dynamical systems, namely the symbolic transfer entropy T^y introduced in [|l6ll . Thus, using the 
asymmetry of the CMI under the interchange of the time series, one can easily construct indices 
of information flow, for instance the difference T^y ~ '^yx- 

Now, we introduce and motivate the use of a new coupling directionality measure based on the 
mutual information of transcripts defined as follows, 

I(Ty,a, Tp^a) = ^(Ty,a) - H{Ty^a I Tp,a)- (14) 

First, note that Eq. (IT4l) is only a function of transcripts between symbolic representations. Fur- 
thermore, it displays the same invariance under the interchange of 7 andyS and asymmetry when 
interchanging the roles played by a and yS as Eq. (fT3l) . Having in mind that transcripts account 
for the relationship between symbolic representations, one can discover qualitative similarities 
between Eqs. (fT3l) and (fT4l) . In fact, one observes that stronger (weaker) dependence between 
P and 7, increases (decreases) both informations given by Eqs. (1131) and (fT4l) . However, a rele- 
vant difi"erence is evident in Eq. (IT4l) . i.e. the estimate of information flow is calculated in lower 
dimension. 

Let us assume again that {7,} = {cr,+A} and consider the case {yS,} independent of {a,} and {7;}. 
Clearly, 7(7, jS | a) = in this case. We are going to show next that the same property holds 
for I{Ty^a, Tj3,a) undcr the additional assumption that a (hence 7) or yS are uniformly distributed. 



Indeed, using that C{y, a,/3) > 0, Eq. (fT4l) can be bounded as (see (fT2l) with m = 3) 

< Hijp,,) + Hijy,,) + min{//(r), //(a), //OS)} - H{y, a,/3). 

Here, //(y) = H(a) and H(y,a,/3) = //(jS) + H(-y,a) since we assumed independence. The latter 
expression can also be written as H{y, a,/3) = H{a,/3) + H{y, a) - H(a). Therefore, 

liTya, Tfsa) < H{Tp^,) + //(t,,J + min{//(a), Hm - H{a,/3) - H(y, a) + H(a). (15) 

Using Eq. ([5]), H(a,/3) = H(Tfja,/3) = HiTpa, a) and H{y, a) = H{rya, a). Let us assume now that 
the variable jS is uniformly distributed. Then, mm[H{P), H{a)} = H{a) and H(a,/3) = //(t^,^, a) = 
HiTp^a) + H{a), where in the latter expression we used again the independence of a andjS. Thus, 
inequality (fT?)) becomes 

I{Ty,a, T/j^a) < H(Ty^a) + H{a) - H^Ty^a, »)■ (16) 

Similarly, if the variable a is uniformly distributed then mm{H(J3), H{a)} = H(J3) and H(a,/3) = 
H{Ti}^a,P) = H{Tfj^a) + H(J3). Replacing these equations in (fTSi) . we obtain again Eq. (fT6l) . It should 
be noted that the right hand side of (fT6l) is independent of the variable /3. As shown below, distri- 
butions closer to the uniform distribution can be obtained by a suitable choice of the parameter T. 
In addition, in case of independence the upper bound in Eq. (fT6l) can be made negligible using a 
convenient relation between T and A. 

The selection of embedding parameters is a common problem which has been extensively 
discussed in the field of non-linear systems [25]. Directionality measures are not the exception 



111 811 . We present in the following an example intended to show the dependency of the direction- 
ality measures (fT3] ) and (IT41) on the parameter T (time delay used to generate the ordinal pat- 
tem) for constant L = 4. Consider the following bidirectionally delay ed-coupled logistic map 
/ : [0, 1] — > [0, 1], f{x) = 4x(l - x) defined by the equations 

x{t) = figy^x mod 1), with 

gy^, = kiy(t - Ai) + (1 - ki)xit - 1), 

y(t) = f(g,^y mod l),with (17) 

g_,^y = k2x(t - A2) + (1 - k2)y(t - 1), 

where Ai = 5 and A2 = 2 are the coupling delays, and ki 6 [0, 1] and ^2 6 [0^ 1] are the coupling 
strengths. We investigate the coupled logistic map (fTTl) for the coupling parameters ^2 = 0-2 and 
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FIG. 1. Upper row: Conditional mutual informations /(a,+4,jS; | or,) (solid curve) and I(fii+4, a, | j6,) (dashed 
curve) versus ky. Lower row: The mutual informations /(Tq, ,t^,q,) (solid curve) and I{t„ ,Tafi) (dashed 
curve) versus k\, where Tq, a, = a,+4, tI jS, = yS;+4, and rgaO^/ - A- Different panels show the behavior of 
the coupling directionality measures (Eqs. (IT3T l and dHJ) for different values of T. All results were obtained 
for the coupled logistic map (ITT]) using L = 4, A = 4 and times series of length N = 10^ data points. 



ki e [0, 1] as in reference [|l8ll . Let {a,}, {/?,} be the symbolic representations of the time series {xi}, 
{yi}, respectively. For every value of ki, we have evaluated the measures defined in Eqs. (IT3T ) and 
((14)) for several time delays T and time lags A e [-10, 10]. Typically the response of the coupling 
directionality measures displays a maximum for a certain value A = A^. For this system, A^ = 4 
leads to a good description of the information directionality US\\ . 

Figured] shows the behavior of the coupling directionality measures (IT3]) and ([14]) versus ki for 
different values of the time delay T. In general both measures are able to describe correctly the 
overall coupling directionality. In fact, we observe that for ki < 0.2 the direction of information 
is X ^> y, but a crossover to y -^ jc is observed when increasing the coupling constant ki, as 
expected from Eq. ([T7] ). Note that the solid (dashed) curves in Fig. [T] describe the information 
flow y ^ X (x ^> y), respectively. However, particular details are observed for different values of 
the delay time T. Here, Tq, and rl denote the transcripts between ordinal patterns of the same 
symbolic representation at different times, as explained in the caption of Fig. \T\ For T = 1 and 
ki = 0, /(t^^\ Tyj^or) (solid curve) displays a bias to positive values, while I{ai+4,j3i \ a,) ~ 



(solid curve), as expected. For increasing ki, both /(r^^^r^.a) and I{ai+4,/3i \ a,) increase rather 
monotonically, except around the value ki ~ 0.20. For ki < 0.20 both I(fii+4, at \ yS,) and I(j"^\ Tafi) 
(dashed curves) indicate the right direction of information flow, but for increasing k\ , I{fii+4 , or, | jSi) 
displays a strong unexpected increasing trend. In contrast, /(rl , Tafi) (dashed curve) remains 
rather constant. 

For T = 9, 1(ai+4,j3i \ a,) and lir^^K Tp^a) describe correctly the coupling in the direction y ^ x. 
It should be noted that for this value of the delay time, /(t^\ t^,«) ~ for k^ = 0. However, 
Iifii+4, Ui I Pi) (dashed curve) provides a poor description of the coupling directionality, displaying 
an even stronger trend than that observed for T = 1. On the other hand, /(tL , Tafi) provides a 
better description, but still displaying a weak increasing trend for larger k^. For T = 27, both 
measures provide the same description of the coupling directionality in the system and can rather 
be distinguished by eye inspection. In fact, we demonstrate below that under certain conditions 
both coupling directionality measures are identical. 

Let us assume that min{//(Q'), H{fi)} = H{a) and that the following relation 

C(a,y8,r) = C(a,r) + C(a,A (18) 

holds for a particular choice of the embedding parameters L and T. For {y,} = {Qr,+A}, Eq. (jTSl) 
indicates that the coupling complexity of the three symbolic representations can be expressed 
as the sum of two terms, namely an "auto"-coupling complexity C{a, y) and a "cross"-coupling 
complexity C{a,/3). Using Eq. (fTS]) one obtains 

H(a, 7) - Hia) - Hia,/3, y) + H(a,/3) = H(t^J + H{Tp^,) - Hijy^,, t^,„), (19) 

which immediately implies the equality of Eqs. (fT3l) and (fT4l) . Thus, we have demonstrated that the 
CMI estimator can be reduced to the mutual information of transcripts when Eq. (ITST ) is fulfilled. 
The dimensional reduction can be very important in time series analysis because the number of 
A^ joint symbols grows exponentially with A^, while the length of real- world time series is finite. 
Therefore, the use of expressions similar to Eq. (|T4) may in some cases prevent from undersam- 
pling and, in any case, it improves the statistical significance of the estimations. 

Another interesting condition which deserves special attention is C{y, a,/3) = 0. This particular 
case is relevant for a wide range of systems, where a low complexity can be achieved by generating 
symbolic representations using a suitable time delay T. Typically, the dependence of C on T is 
such that C(T) decreases when T grows. This condition can be compared to that of maximizing 



the sorting entropy [|20ll already discussed in [|l8ll . As before, let us consider {y,} = {a,+A}, with 



A > 0. The coupling complexity C(y, a,/3) can be written as follows (see Eq. (fT2l) ') 

C{y,fi, a) = min{/(Q;; Ty^a, T/j,a), I{fi\ Ty^a, T/3,q.)}. (20) 

Furthermore, Eqs. (|6) and (|7} imply that the entropies H(y, a,/3), H{a, Ty^a, Tp,a), and //(yS, r^ q., t^ q.) 
are identical. According to Eq. (I20l ). the variable leading to the minimum mutual information 
(C = in this case) is independent of the joint transcript variable (t^q,, Tp^a)- Let us assume that 
mm{H{a), H(J3)} = H{fi). Then, the joint entropy of the three symbolic representations can be 
written as 

H{y,p, a) = HQ3) + H(Ty^,, r^, J. (21) 



We will invoke now the property of monotonicity of the coupling complexity [12311 . In fact, one can 
demonstrate that if mm[H{a), H(J3)} = H(J3) then C(y, a, (5) > C{y, a), which leads in this case to 
C(y, a) = 0. Thus, monotonicity implies the independence of the variables a and Ty^a- Similarly 
to Eq. (|2T1) . the following conditions hold 

H{y,a) = H{a) + H{Ty,„) 

Hia,/3) = H(J3) + H{Tp,a). (22) 

where Eq. (|2^ follows from the independence of the variables P and t^ „. Using Eqs. (I2T1) and 
(1221) . Eq. ^ becomes 

I{y,l5 I a) = H(y, a) + H(J3, a) - H(y,/3, a) - H(a) = H(ry,„) + H{Tp,^) - H(t^,„, t^, J, (23) 

which implies the equality of Eq. (fT3l) and Eq. (fT4l) and thus dimensional reduction. In case 
mm{H{a),H{fi)} = H(a), the property of monotonicity has a more general implication, i.e. 
C(y,a,/3) > C(y,a) and C(y,a,/3) > C(J3,a). Using these conditions, one can analogously 



derive Eq. (|23T ). The property of monotonicity is proved for the multivariate case in [|23ll . 

We have just shown that the coupling complexity is a relevant quantity to take into account 
when analysing coupling directionality. In the next example, we monitor the behavior of C and 
other information measures versus the delay time T. We consider again the coupled logistic map 
(fTTl) and generate symbolic representations {or,}, {jS,} for the time series {x,}, {j,} and coupling pa- 
rameters ki = 0.6 and kz = 0.2. In this example {y,} = {at+i}. Figure |2] shows the behavior of 
different information measures as a function of the delay T used to generate ordinal patterns. Fig- 
ure I2a) displays the complexity C{y,fi, a), and the complexities for the pairs C(y, a) and C(J3, a), 
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FIG. 2. a) The complexity C versus the delay T. The solid curve indicates the complexity C{y,p,a), 
while the dotted curve and the dot-dashed curve display the complexities C{y, a) and Cifi, a), respectively 
(more details in text). The inset shows the mutual information I{a,/i)) versus T. b) The solid curve shows 
the entropy of transcripts H{Ty^a,T^^a), the dotted curve the conditional entropy H{y,p \ a), and the dot- 
dashed curve the conditional entropy H{y,p \ /?) versus the delay T The difference between the conditional 
entropies cannot be observed due to overlapping, c) The solid curve displays the entropy //(t^,^), the dotted 
curve the conditional entropy H(Ji \ a) and the dot-dashed curve the conditional entropy H{a \ P). All 
results were obtained using L - 4 and M = 2^^ data points. 



evaluated using Eqs. (fT2)) and ([8]), respectively. We observe that the complexity C(y,/3, a) is always 
larger than any of the complexities for the pairs. In addition, this plot shows that all complexities 
approach zero for delay T > 15. Thus, requesting C{y,/3,a) ~ for the highest dimension auto- 
matically warranties the same condition for lower ones. The inset in Fig. |2ta) shows the mutual 
information of the symbolic representations I(J3, a) versus the delay T. For this coupled system, 
I(a,/3) decreases for increasing T as well. Figure l2lb) andlZic) show the behavior of the entropies 
associated with transcripts and the conditional entropies. For this system, it is hardly possible to 
distinguish between the conditional entropies. More important, we observe in both plots that for 
C{y,/3,a) ~ 0, the conditional entropies approach the value of the entropy of the transcripts as 
predicted by Eqs. ^ and (1211) . 

We turn now the focus to the comparison of the two coupling directionality measures (Eqs. (fT3]) 
and ([14)) ') within the regime (C ~ 0). To this end, we discuss in more detail the coupled logistic 
map (flTl) for delay time T = 27 (right column in Fig. [D. Figure |3la) shows the symbolic transfer 
entropies for both coupling directions, x — > j and j — > x, versus the coupling parameter ki . For 
ki = 0, there is no information flow y ^> x but a clear response is observed for the information 
flow in the opposite direction, as expected. For ki < 0.3 the response is non-monotonous for both 
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FIG. 3. a) Conditional mutual informations I{ai+4,fii \ a,) (solid curve) and IiPi+4, a, | yS,) (dashed curve) 
for the coupled logistic map defined in Eq. ( fTT] ). b) The mutual informations I{t[, , Tp^a) (solid curve) and 
KTa , Tafi) (dashcd curve), where r^, 'a,- = Q',+4, t„ /3i = /3,+4, and Tjs^aO^i - /?,•. c) The difference I{ai+4,Pi \ 
a,)-/(TQ, , T/j^a) indicates the error when using Eq. (IT4] |. d) Idem upper right for I(J3i+4, ai \ I3i)-1{T^ , Tafi). 
All results were obtained using L = 4, T = 27 and times series of length N = lOr* data points. 



directions probably due to the dynamical features of this coupled system jlSl] . In particular the 
crossover point, which is expected to occur around at ki ~ 0.2 is slightly shifted to higher values. 
For k\ > 0.3, the information flow y ^ x increases monotonically while the information flow 
X ^ y remains almost constant. It should be remarked that these results can only be compared 
qualitatively with those presented in reference [llSll . since the evaluated measures are different. 
Figure |3lb) shows the mutual information between transcripts as described in the caption. As 
mentioned above, it is hardly possible to find a difference by eye inspection between the upper 
left and lower left panels. The difference between conditional mutual information and mutual 
information of the transcripts (Eqs. (ITJI ) and (fT4] )) is quantified in Figs.[3tc) and[3td). The mean 
and standard deviation of the difference are around 3.5x10"^ and 1.1x10"^ in both cases. 

As a second example, we present two linearly bidirectionally coupled autoregressive models 
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FIG. 4. The symbolic representations |a,) and {/J,} correspond to time series x and y for the coupled 
autoregressive models defined by Eq. (|24] |. a) Conditional mutual informations /(a,+i,/3,- | a,) (solid curve) 
and I(J3i+i,ai \ /3,) (dashed curve), b) The mutual informations /(Tq, ,T/j^a) (soUd curve) and I{t„ ,Tafi) 
(dashed curve), where Tq, a,- = a,+i, To Pi - /3i+[, and Ta^/jai = Pi. c) The difference I{ai+[,Pi \ a,) - 
/(Tq, , Tjs^a) indicates the error when using Eq. (IT4] |. d) Idem upper right for /(/3;+i , a,- | Pi) - I{Tg , Ta,ii)- AH 
results were obtained using L = 3, T = 30 and times series of length N - 10^ data points. 



defined by the following expression, 



Xi+i = kiXi + kcji + ?7,^ ji+i = k2yi + kxi + rj]. 



(24) 



where k\ = 0.6 and ^2 = 0.5, and ri^ and rf"' are normal random numbers. The parameters k^ = 0.2 
and k are the couplings between system components, where k is varied in the range k e [-0.6, 0.6]. 
This system was studied analytically using transfer entropy in il6\ for the case kc = 0. As before. 
Fig. 13 a) shows the CMI for both coupling directions x ^ y and y ^ x versus the coupling 
parameter k. The solid curve indicates that the information flow y ^^ x never vanishes. This is 
expected since kc = 0.2 for the whole range of k values. A clear asymmetry is observed between 
the regions A; > and k < 0, since the symmetry of Eq. (|24l) is broken for kc 4^ 0. Thus, the CMI 
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T^l^at = Pi-i, and T^pj^at-i = fit. c) The difference I{ai+4,/3i \ ai,Pi.i) - /(t^'^^tJ'^ | T^p indicates the 
error d) Idem upper right for I(fii+4, at \ Pi, a,-i) - /(t1 , Tq, | t[1). All results were obtained using L = 3, 
T = 30 and times series of length N - 10^ data points. 



/(q',+i,A- I ad > m^i,ai I A) for -0.25 < k < 0.20 and /(ff,+i,A I «,) < /(>e,+i,Q'; | A) for 

-0.25 > fc > 0.20. For k ~ 0.20 and k 0.25 the values of the CMI are similar, revealing a 

balanced situation with no preferred coupling direction. It should also be noted that /(/3,+i, a, | /3;) 
vanishes for k = since there is no information flow x ^> yfor this value of the coupling parameter. 
FigureSfb) shows the mutual information between transcripts, as described in the caption of Fig. H] 
Once again, there is a striking similarity between the left panels. Figure Hfc) andH^d) indicate the 
difference between the two approaches. In both cases, the mean and standard deviation of the 
difference are around 9x10""* and 1x10""^. 
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B. Generalization for more conditions 

We return now to the discussion of Eq. (fT3] ) and consider first the case where the condition 
expresses the joint information of two processes, as follows 

m 7 I «,yS) = H(e, a,/3) + H(y, a,/3) - H(e, y, a,/3) - H(a,fi). (25) 

where the CMI has been written in terms of Shannon entropies. Here, we will restrict ourselves to 
the bivariate case and find the generalized form of Eq. (fT4T i when accounting for more conditions. 
For instance, in Eq. (|25] ) we can assume that {0,} = {ai+^i } and {7,} = {jSj+aJ with Ai > A2 > 0, thus 
the variable ia,/S) accounts for the joint past of the coupled processes. We use again the condition 
C(6, y, a,P) = 0, here for four variables, and write as before C in terms of mutual information as 
follows 

0(9, y, a,/3) = min{/(a; Te,„, r^,;?, r^^f^), I(J3; Tg^^, Ty^, t„,/j)}. (26) 

In the limit of vanishing coupling complexity, Eq. (|26|) implies that the variable associated with 
the minimum entropy, i.e. a or /3, is independent of the joint transcript variable {Tg^a, Tyfi^ Tafi)- 
In this case, one only needs to invoke monotonicity (see MM) and to follow the same reasoning 
which led us to Eqs. (|2T)) and (|22)) to derive 



I{e, y I a,p) = H{Te,a, r^,/?) + HiTy^p, T„,/j) - H{Tg^a, Ty,/J, T^^p) - Hir^^p) = I{Tg^a, Tyfi I Tafi)- (27) 

Thus, the CMI for two conditions is reduced to one of three transcripts, where Tq,^ accounts for 
the joint conditional process. Following this strategy, one can easily infer that for m conditions the 
analysis can be reduced to one of m - 1 conditions, where only transcripts among symbolic repre- 
sentations are involved. The structure of this approximation scheme naturally induces us to ask for 
further dimensional reduction. From the point of view of the construction, this is always possible 
since the scheme does not diff"erentiate between ordinal patterns and transcripts. However, one 
has to have in mind that every additional dimensional reduction is performed under assumptions 
different from that expressed by C ~ 0. Thus, it is expected that error increases when reducing the 
dimensionality of the problem. However, for some of the considered systems, we have observed 
that further dimensional reduction still renders very good approximations which describe the main 
features of the coupling directionality. 

As an example of the application of Eq. (|T7l) . we consider once again the coupled logistic map 
(fTTl) already analyzed using Eq. (fT4l) . but we include an additional condition to account for the 
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FIG. 6. a) Conditional mutual informations I{ai+i,fii \ a,) (solid curve) and /(/3,+i,a,- | yS,) (dashed 
curve) for the two linear coupled autoregressive processes defined in Eq. (I24l) . b) The mutual informations 



/(Tq, , Tfj^a) (solid curve) and /(t1 , Ta^js) (dashed curve), where Tq, or,- = a,+i, t„ fit - jS;+i, and Tf^yjor,- - /?;. 



c) The difference I{ai+i,/3i \ or,) - /(r'^ , rgQ,) indicates the error when using Eq. (IT4] |. d) Idem upper right 



JA) 



for I(fii+i,ai I /3i) - I{t\ ', Tafi). All results were obtained using L = 4, T = 30 and times series of length 
N = 10^ data points. 



joint past of the processes. Figure |5]is similar to Fig. |3]but the compared measures have the 
form of those in Eq. (|27T ). Figure |5]reveals that including the joint past as condition in the CMI 
improves the characterization of the coupling directionality displaying a more sensitive response 
within the range of coupling values where crossover behavior occurs (k ~ 0.2). The accuracy of 
our approach can be observed in Figs. |5jc) and|5jd), with a mean and standard deviation around 
4x10-3 and 4x10-^ 
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C. The influence of dimensionality 



A comparison of Eq. (|T4|) and Eq. (1271 ) indicates that the space dimension to estimate informa- 
tion flow increases with the number of conditions. In general, the CMI requires the calculation of 
the entropy of the m-dimensional joint process, where m is the number of symbolic representations 
involved in the calculation. In addition, the number of available states in this space grows with L 
as (L!)'". Then, the curse of dimensionality becomes an issue to obtain reliable estimates and one 
has to find a suitable compromise between m, L and the length A^ of the time series. Since the right 
hand side of Eq. (O and Eq. (|27l) imply dimensional reduction, they may provide a more accurate 
quantification of the coupling directionality. 

To investigate the influence of dimensionality, we have considered the autoregressive models 
defined in Eq. (|24l) but using k^ = for the sake of simplicity. Figure |6] shows the same measures as 
in Fig. [3]but evaluated for L = 4 and using the same number of data points. The symbolic transfer 
entropies (Fig. IMa)) clearly unveils the effect of increasing dimension. In fact, one expects that 
the information flow y ^> x vanishes in this case. However, the solid curve, which indicates the 
information flow y ^> x, displays an approximately constant value higher than zero due to poor 
statistics. On the other hand, our estimate expressed by Eq. (fT4l) is more robust against increasing 
dimension, since the dashed curve is still very close to zero mutual information, as observed in 
Fig.[6tb). In this case, the difference between the two coupling directionality measures displayed 
in Figs.[6tc) and[6td) is larger because of poor statistics as well. 



D. Other approaches 

Some authors have considered approaches to describe coupling directionality using ordinal 
pattems, where the information flow is calculated through the sorting information of future values 



1911 . Some of these information 



among ordinal pattems describing the history of the systems 11181 
measures even consider the use of ordinal pattems of different lengths L. We will show that our 
approach fits in these constmctions and can be implemented in an elegant way. First, we focus on 
the definition of a transcript between ordinal pattems a^' and a^- of lengths Li, and L2, where we 
assume L\ > L2 without loss of generality. Since Sl2 c S^ then every element in Sl2 can also be 
expressed as an element of the larger group <Sz,, . Let AL = Li - L2 be the difference between the 
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length of ordinal patterns. Within Sl, , the symbol a^- can be expressed as follows 

a^^ = (aJ^ ■ ■ ■,a'll_,,L2,L2 + 1, • • • ,L2 + AL - l) (28) 

By means of this procedure it is always possible to evaluate transcripts between ordinal pat- 
terns of different leng th. Note that the group embedding defined by Eq. (|28]) conserves the 



transcript scheme [|21ll of the smaller group. Let [xt} be a time series and consider the symbol 
^n ~ \^0' ■ ■ ■ ' ^L~i) which describes the rank ordering of the sequence (.x:,,-l+i, ^f,-L+2, • ■ • , -^n)- 
The sorting of the value Xjj+a with A > can be expressed in terms of transcripts using Eq. (|28T l 

To{a'^,...,al„L) = a^;\ (29) 

where a^"^' describes the rank ordering of the sequence {xt,-L+i, Xt^-t+i, ■ ■ ■ ,Xt,, Xi^+a) (for simplic- 
ity we assumed T = 1). Thus, the transcript t accounts for the sorting information of the new value 
among the sequence of the previous ones. As an example, we apply these concepts to the momen- 
tary sorting information transfer (MSIT) introduced in [llSll . This measure was chosen since other 
approaches considered in the literature are special cases of the MSIT jlSl] . Let us consider first the 



momentary information transfer defined as |18|l 



/J^;^(A) = _^M^„3;,^A,z)log- 



p{xt,y,+A \z) 



p(xt I z)p(yt+A I z) 
with the condition z = (x^J^ ,y^_^'^_^^). (30) 

Here Xt and yt+A are values of the time series, x^J^ and jy+X-i ^^ delay vectors of length M^^ and 
Mj.^, which determine the joint past z of the dynamical systems. The momentary sorting information 
transfer (MSIT) is derived from Eq. (I30l) when only accounting for sorting information of yt+A 
among yy_|_~^_j and Xt among x^J^ [18]. This quantity can be written in the form of a CMI as 

/J1?^^(A) = m^A,7i I «,+A-i,A-i), (31) 

where 0,+a, y,-, a,+A_i and jSi^i are the ordinal patterns for (yi+A-M,,+\,yi+A-M,,+2,- ■ ■ ,yi+A), 
{Xi-M:^+\,Xi-M,^+2,''',Xi), (yi+A-M,^.,yi+A-M.^.+i, • • • ,yi+A-i), and (xi-M,j.,Xi-M.^+i,' • • ,Xi-i), respec- 
tively. Then, it is clear that for A > 0, A = jc and B = y, and for A < 0,A = y and B = x. 

We immediately identify that our approach as given in Eq. (|27] ) can be applied to the MSIT as 
follows 

/^B^'^(A) - IiTo,a, Ty,p I T„,^), (32) 
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FIG. 7. a) Conditional mutual informations I{6i+5,yi \ Qf(,_i)+5,j6,_i) (solid curve) and I{6i-2,yi \ 
Qr(,_i)+5,yS;_i) (dashed curve) for the coupled logistic map defined in Eq. (ITT] ), b) The CMI I{Tyl, Tjjj \ Tq-^j) 
(solid curve) and I{Tf^^,Ty^p \ T^fi) (dashed curve), where T|,^ja(,_i)+5 = 6i,+5, T-y,/3/3,-i = 7h tI^^Jcko-d-i = 
6j-2, and Ta,fia(i-\)+5 = Pj-i . The ordinal patterns 6 and y are of length Li = 3, and a andyS of length L2 = 2. 
Thus, all patterns were embedded in S^. We used T = 30 and times series of length N = 10^ data points. 



M,, 



where the transcripts Tg^a, Ty^ which provides the sorting information of x, among x ." and j,+a 



M-, 



among y,^X-i ^^ evaluated according to Eq. (|29] ). The transcript Tq,,^ corresponds to the joint past 
and is evaluated in the general case using the group embedding defined in Eq. (I28l) . It should be 
noted that the approach given by Eq. (|32l) is not restricted to the use of consecutive values for 
generating ordinal patterns. In fact, one can always search for a suitable delay T satisfying the 
condition C ~ 0. We applied the above described approach to the coupled logistic map (Eq. ([TT]) ') 
using the same coupling parameters as before, for the sake of comparison. We have chosen ordi- 
nal patterns of length Li = 3 and L2 = 2, thus all ordinal patterns are embedded in S^. Since the 
purpose here is to test the approximation given by Eq. (|27l ). a delay time T = 30 has been used to 
generate ordinal patterns and satisfy the condition of vanishing complexity. For the joint condi- 
tion (a,/3), ordinal patterns were generated according to Eq. (|28) and the transcripts according to 
Eq. (|29l ). We have considered values of A in the range A e [-7, 7] but we show results only for the 
A values leading to the maximum response for every direction, namely A = 5 and A = -2. Fig- 
ure |7] presents a comparison of the two measures appearing in Eq. (I32l) . The agreement between 
M^B ^^'^ I{T6,a,Tyfi I ^a,^) IS also remarkable for this approach. The mean value of the error 
calculated over the different values of k^ is around 5 x 10"^ with a standard deviation of 3 x 10"^. 
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FIG. 8. a) The complexity C versus the delay T for the frontal electrode pair F4-FP2 in the pre-ictal state. 
The solid curve indicates C(y,/i, a), while the dotted curve and the dot-dashed curve display the complex- 
ities for the pairs (y, a) and {a,/3), respectively (more details in text). The inset shows the behavior of the 
mutual information I{a,/i) versus T b) The solid curve shows the entropy of transcripts H{Ty^a,T/s,a), the 
dotted curve the conditional entropy H{y,p \ a), and the dot-dashed curve the conditional entropy H{y,p \ p) 
versus the delay T. c) The solid curve displays the entropy H(Tafi), the dotted curve the conditional entropy 
H{P I a) and the dot-dashed curve the conditional entropy H{a \ P). Results were obtained using L = 4 and 
M ~ 10^ data points. 



These results are in perfect agreement with those reported in [j 180. 



E. Application to real world data 



We analyze the electrical brain activity of an infant patient suffering from frontal lobe epilepsy 
(FLE). It should be remarked that it is not the purpose of this work to perform a clinical study but 
to demonstrate the applicability of the above presented methodology to an example of real world 
data. A clinical study of the evolution of the brain electrical activity during therapy has already 
been presented in Bunk et al. nlin . 

The EEG recording was acquired during a time interval of 15 minutes at a sampling rate of 
250 Hz and a signal depth of 16 bits, and consists of 21 synchronously obtained time series. 
The positioning of the electrodes followed that of the standardized 10-20-Intemational System of 
Electrode Placements. We consider an EEG recording which documents a seizure and perform the 
information directionality assessment for the pre-ictal and ictal states separately. 

Figure [8] shows the behavior of some information measures evaluated for the EEG pair F4-FP2 
in the pre-ictal state as a function of the delay T used to generate the symbolic representation. 
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FIG. 9. a) The solid curve displays the CMI for the EEG recorded at FP2 and F4. {of,} and {yS,} are the 
symbolic representations of the time series obtained at F4, FP2, respectively. The symbolic sequence {y,} = 
{a,+A), where A e [-0.4 sec, 0.4 sec]. The dashed curve shows the mutual information of the transcripts 
{(Ta,y)i] and {iTafi)i}- Both mcasurcs were evaluated in the pre-ictal state, b) Idem a) for {or,} and {yS,} 
corresponding to F3, FPl, respectively. Both measures were evaluated in the pre-ictal state, c) Idem a) 
but in the ictal state, d) Idem b) but in the ictal state. Results were obtained using the parameters L = 3, 
T = 1.2 sec and time series of length M ~ 10^ and M ~ 1.3x10^ data points. 

Here {a,}, {j0,}, {y,} are the symbolic representations of the time series {Xi} of F4, {j,} of FP2, and 
[xi^i }, respectively. All measures except the mutual information I(a,/3) behave as in Fig.|2l In fact, 
I{a,fi) displays exactly the opposite trend, asymptotically approaching a saturation value greater 
than zero. It is remarkable that all approximations given in section |Il] are valid even though the 
I{a,/3) unveils completely different interactions. According to Fig. |8la), we generate ordinal pat- 
terns using a T value to satisfy region (C ~ 0) and calculate for every pair of electrodes and for 
every state the measures appearing in Eq. (fT?] ). where {y,} = {ff;+A}. These information direction- 
ality measures were evaluated for different time lags A, in order to determine the main driving 
electrodes and the lag of the maximum response. 

Figure |9] shows the CMI and the mutual information of the transcripts for the EEG pairs FP2- 
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F4 and FP1-F3 in the pre-ictal and ictal states. These EEG pairs were chosen since they lead to 
the strongest responses. All plots display a maximum for positive Amax values, clearly indicating 
that FP2 and FPl are the driving signals. We observe that both measures provide almost the same 
information about the coupling directionality. In particular, both curves indicate the same position 
for the maximum response A^ax- Within the covered range of A values, the error is rather constant 
(~ 4x10"^), except around A = where lower values are observed. This shows indirectly the 
weak dependence of C on A for this real world data. In all cases, the mutual information of the 
transcripts displays lower or equal values than the CMI. 

A global analysis considering all pairs shows that for the pre-ictal (ictal) state 17 (14) out of the 
20 strongest responses are driven by frontal signals. This result agrees with the brain pathology 
of the infant and suggests that signals from the epileptic focus might be driving other brain areas 



ll27n . A comparison of Figs. |9ta) and |9tb) with |9lc) and Hd), indicates that for the ictal state 
responses increase and A^ax becomes longer. For the pre-ictal state, the mean lag < A^ax >= 
0.041 + 0.014 sec, while for the ictal state < A^ax >= 0.061 + 0.017 sec, where averages were 
taken over the 20 strongest responses. 



IV. CONCLUSIONS 

The concept of transcripts arises naturally when studying relationships between dynamical sys- 
tems using ordinal symbolic dynamics. Using transcripts one can exploit properties of the sym- 
metric group and combine them with information theoretical approaches. In this work, we have 
considered the problem of estimating coupling directionality for the bivariate case, and introduced 
novel information directionality measures which depend only on transcripts for single and joint 
conditions. Generalizations of these information directionality measures to the muti-variate case 
are feasible and will be presented elsewhere. These new directionality measures have the im- 
portant property of calculating the information flow estimate in lower dimension, which may be 
preferable for small data sets. We have also proved that the well established conditional mutual in- 
formation quantifiers reduced to the proposed measures when a condition of vanishing complexity 
is fulfilled. A rather general search strategy for low complexity has also been provided. 

Furthermore, we have introduced the concept of group embedding which allows generalizing 
the definition of transcripts to ordinal patterns of different lengths. Using this extension, different 
approaches to calculate information flow could be considered within the same framework. We have 
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applied our method to synthetic model data and real world data as well. An example was presented 
demonstrating the suitability of this transcript based approach to tackle information directionality 
in EEG data as a diagnostic tool. 
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